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ABSTRACT 

This validation study investigated the instructional 
effectiveness of seven programmed booklets on selected topics in drug 
education against the criterion of student achievement. A 
post-test-only control group design was used. Six of the seven 
booklets were found to be ins tructionally effective as measured by 
objective referenced tests. The problems of doing this type of study 
are discussed as an aspect of the work of an instructional 
development agency's formative evaluation staff. (Author) 
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Abstract 



This validation study investigated the instructional effective- 
ness of seven programmed booklets cn selected topics in drug educa- 
tion against the criterion of student achievement. A Post-test-only 
control group design was used. Six of the seven booklets were found 
to be instructionally effective as measured by objective referenced 
tests. The problems of doing this type of study are discussed as an 
aspect of the work of an instructional development agency's formative 
evaluation staff. 
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An Experimental Validation 
Of Seven Programmed Instructional Booklets 
For a Course on Drugs 

William L. Holzemer 
Syracuse University 

The? developer of programmed instruction is reguired to initiate 
the empirical validation of his materials to determine their instruction- 
al effectiveness. Yet the process of validation frequently becomes so 
complex that the developers often retreat from the task because of con- 
straints of interest, time, and money. Despite the recommendations of 
the Joint Committee on Programmed Instruction (1966) and others (Glaser, 
1966; Komoski, 1966; Popham, 1970), the developer's responsibility for 
validation has often been ignored or shifted to a publisher or buyer. 

Popham (1970) stated that "validation refers to the accumulation 
of empirical evidence regarding the success of given materials in 
promoting attainment of their particular instructional effectiveness" 
(p. 217). The four types of empirical evidence considered most im- 
portant are (1) changes in achievement, (2) arount of time to com- 
pletion, (3) percentage of errors for each frame (in linear programming), 
and (4) student attitudes (Jacobs, 1966). The one category which is 
generally neglected in validation studies is achievement, and conse- 
quently achievement is frequently omitted as a criterion for selection 
(Garner and Zernp, Jr., 1971). These representative problems of the 
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developer are illustrated in the validation of programmed materials de- 
scribed below. 

The School of Social Work in conjunction with the Center for In- 
structional Development at Syracuse University developed an interdisci- 
plinary course, Drugs in Perspective. The course is described in the 
Stud ent Manual in the following manner: 

Drugs in Perspective is designed to provide you with a 
broad, objective knowledge base in the area of drugs and their 
use in contemporary society, allowing you to examine your own 
attitudes as well as others in relation to drugs. The course 
is divided into the following modules: Defining the Drug Pro- 
blem, Pharmacological Aspects of Drugs, Drugs and the Law, 
and Major Treatment Approaches. 

Drugs in Perspective utilizes various educational tech- 
niques, including role playing, self-instructional booklets, 
slide/tape presentations, and simulation exercises, while 
allowing you to move at your own pace. 

The course which is open to all students (freshmen through 
graduate students) is part of a total drug education project 
utilizing community and University resources. (Higley and 
Eickmann, 1973, p. 3). 

Eleven topic-specific branching programmed instructional booklets were 

produced for the Module, Pharmacological Aspects of Drugs. 

The booklets were carefully researched by graduate students in the 
School of Social Work. Two consulting faculty in neurochemistry and 
neuropharmacology from Upstate Medical Center, State University of 
New York, approved the booklets 1 content. Three community agencies 
dealing with drug prevention, education, and crises also examined 
and approved the booklets. Students' affective responses to the book- 
lets were examined with a short six-item questionnaire and interviews. 

At the request of the project director and the development staff, 

a validation study was initiated to examine student achievement. Only 
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seven of the eleven booklets were initially examined in order to establish 
acceptable power in the statistical analysis. The seven booklets simul- 
taneously examined were titled* Amphetamines, Barbiturates, Caffei ne, 
Cocaine, Drug Definit ions, Marijuana, and Tranquil izers . 



*See appendix for a list of Staff for Drugs in Perspective and a list of 
authors for the booklets. 
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Design 

A post-test-only design (Campbell and Stanley, 1963) was chosen 
because of its strong internal validity and its applicability to 
existing conditions. Table 1 presents a schematic of the post-test- 
only design. 



TABLE 1 

POST-TEST-ONLY CONTROL GROUP OESIGN 

Experimental R X 0^ 

Control R 0 o 

R = randomization 

X = treatment 

0, and 0 9 = post-test 



Subjects 

The 41? students enrolled in the course, Social Work 250/550, Druqs 
in Perspective, were the subjects. 212 subjects were matched on sex and 
college class and randomly assigned by pairs to four treatment conditions 
(Caffeine, Cocaine , Tranquil izers , and Marijuana ) , SAT-Verbal scores 
were available for 200 students; these subjects were matched on SAT- 
Verbal, sex, and college class and randomly assigned by pairs to three 
treatment conditions ( Amphetamines , Barbiturates , and Drug Definitions ) . 
Tables 2 and 3 present an analysis of the subjects within treatment 
conditions by sex, college clacs, and SAT-Verbal scores. 
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Procedure 

One 50-minute class period was devoted to the validation study. 
Each student upon entering one of the 12 sections received a packet 
with his name on it. The instructor read standardized instructions 
asking the students to remove an INSTRUCTION SHEET from their packet 
and to follow the instructions. Students were informed that their 
test scores would not be given to their instructors. 

Students assigned to one of the seven experimental groups were in- 
structed to remove the programmed booklet and read it. Upon com- 
pletion of the booklet, they were instructed to remove the objective 
referenced test and complete it. Returning all materials to their 
packet, the students could then leave the room. The control groups 
were instructed to remove the objective referenced test, complete it, 
and return it to the packet. They were then instructed to remove 
the programmed booklet and read it; upon completion of the booklet 
they were allowed to leave the room. The control groups were in- 
structed to read a booklet (corresponding to their post-test) in 
order to minimize the effect of intra-sossion contamination. 

Content Val idit.y 

Each programmed booklet contains spec : learner objectives 

within the text. A graduate student teaching in the course wrote the 

objective referenced test items from these objectives. From Table 4, 

it becomes apparent that the objectives were written with different 

degrees of specificity. For example, each learner objective in the 

booklet Drug Definitions was portrayed by one test item; however, 
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three test items were necessary to portray each learner objective in 
the booklet Barbiturates . The author of the test items carefully arrayed 
the learner objectives with the minimum test items necessary anri these 
in turn were reviewed Dy the researcher. 
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Analysis 

Students' responses were collected in machine-scoreable format, 
and the data were rendered for analysis. Means, standard deviations, 
and t statistics were calculated for the significances of the differ- 
ence between the experimental and control group means for each treat- 
ment condition. 

RESULTS 

The developer of these programmed booklets can be reasonably 
confident that students did learn from the material. The results of 
the validation study (Table 5) show that six of the seven experimen- 
tal group means were significantly different from the cor+rol groups. 

The marijuana booklet did not show a significant difference and 
this can be interpreted in two ways. First, it is quite probable 
that the students knew a great deal about marijuana prior to the 
treatment and that the strength of this knowledge was so great that 
the booklet could not show a significant increase. Upon examination 
of the answer sheets for the marijuana booklet, a second explanation 
becomes plausiHe. Three of the answer sheets were found to be irregular. 
Two students in the experimental group only scored 3 correct responses, 
whereas the mean was 12.4; and one student in the control group scored 
zero, whereas the mean was 11.4. All the students attempted all 19 
questions on the marijuana post- test except these three; they attempted 
9, 18, and 5 respectively. Removing these three scores, the marijuana 
booklet shows a significant difference. It would seem that these three 
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students were uncooperative for one reason or another 

The absenteeism was approximately 40* for the required class period 
for this study. Those results with fewer than 20 students per group 
need to be cautiously interpreted because of the loss of power. 

DISCUSSION 

The recommendations of the Joint Committee on Programmed In "truction 
state that one of the developer's responsibilities is to determine the 
instructional effectiveness of materials. This validation study inves- 
tigated the instructional effectiveness of programmed booklets against 
the criterion of student achievement, and the process of carrying out this 
study brought into question the Committee's recommendation. Having con- 
ducted such a study within the naturalistic situation, the researcher 
is forced to question whether the extreme effort required was really 
worthwhile. 

Many problems were encountered in this validation study. The 
initial concern was to formulate a research design that would provide 
the necessary information for the developer and full ill the criteria of 
internal and external validity for the evaluator. After the research 
design was chosen, major logistical problems urose such as meeting 
press deadlines, obtaining SAT-Verbal scores, and compiling individual- 
ized packets. The Center for Instructional Development made sufficient 
time and resources available to the evaluation staff to conduct such a 
rigorous validation study. It is interesting to speculate about the 
desirability of the information obtained from this study in relation 
to the costs . 



As a result of the extensive development work on the booklets and 
the review of the preliminary evaluation data, the developer and pro- 
ject coordinator were convinced of the instructional effectiveness of 
these materials prior to the invest Ration of student achievement. 
However, they wanted objective evidence to support this intuitive 
claim. The validation study supported the claim of instructional 
effectiveness and, therefore, increased credibility with students, 
faculty, community agencies and prospective \ jhlishers. The costs to 
the Center for conducting the study must be examined in relationship 
to this credibility gain. 

It is difficult to know the benefits of the study for the students 
in the course. One tangible benefit was the class time provided for 
reading one of eleven booklets required normally for out-of-class read- 
ing. Future student populations will benefit because if the results 
had been negative, the development process would have been reinitiated. 
Some students may feel a degree of satisfaction for contributing to 
the development of effective instructional materials. 

The concern for the worth of the study was raised primarily by 
the evaluation staff at the Center. Is it desirable to attempt this 
type of study within an instructional development agency? The amount 
of instructional material available on the market wi thout sufficient, 
stated measures of student achievement suggests that a strong pro- 
fessional obligation exists for the Center to conduct such a study. 
The absolutism of the Joint Committee's Recorrmendation was questioned 
by the evaluation staff because of the required commitment of time 
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and money. However, the increased credibility from the objective know- 
ledge of the materials' instructional effectiveness has caused the 
evaluation staff to support the recommendation with a f u] 1 awareness 
of the complexity of operational i 2 i ng the recommendation. 
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